Compatibility and inclusion of similarity element resolutions

ABSTRACT

For adaptive similarity search resolution in a data deduplication system using a processor device in a computing environment, multiple resolution levels are configured for a similarity search. Input similarity elements are calculated in one resolution level for a chunk of input data. The input similarity elements of the one resolution level are used to find similar data in a repository of data where similarity elements of the stored similar repository data are of the multiple resolution levels.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application, listed as Ser. No. 13/941,999, is cross-relatedto the following seventeen applications each listed as: Ser. Nos.13,941,703, 13/941,873, 13/941,694, 13/941,886, 13/941,896, 13/941,951,13/941,711, 13/941,958, 13/941,741, 13/941,742, 13/941,769, 13/941,782,13/941,982, 13/941,800, 13/942,009, 13/942,027, and 13/942,048, all ofwhich are filed on the same day as the present invention and the entirecontents of which are incorporated herein by reference and are reliedupon for claiming the benefit of priority.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates in general to computers, and moreparticularly to compatibility and inclusion of similarity elementresolutions in adaptive similarity search in a data deduplication systemin a computing environment.

Description of the Related Art

In today's society, computer systems are commonplace. Computer systemsmay be found in the workplace, at home, or at school. Computer systemsmay include data storage systems, or disk storage systems, to processand store data. Large amounts of data have to be processed daily and thecurrent trend suggests that these amounts will continue beingever-increasing in the foreseeable future. An efficient way to alleviatethe problem is by using deduplication. The idea underlying adeduplication system is to exploit the fact that large parts of theavailable data are copied again and again, by locating repeated data andstoring only its first occurrence. Subsequent copies are replaced withpointers to the stored occurrence, which significantly reduces thestorage requirements if the data is indeed repetitive.

SUMMARY OF THE DESCRIBED EMBODIMENTS

In one embodiment, a method is provided for adaptive similarity searchusing compatibility and inclusion of similarity element resolutions in adata deduplication system using a processor device in a computingenvironment. In one embodiment, by way of example only, multipleresolution levels are configured for a similarity search. Inputsimilarity elements are calculated in one resolution level for a chunkof input data. The input similarity elements of the one resolution levelare used to find similar data in a repository of data, where similarityelements of the stored similar repository data are of the multipleresolution levels.

In another embodiment, a computer system is provided for adaptivesimilarity search using compatibility and inclusion of similarityelement resolutions in a data deduplication system using a processordevice, in a computing environment. The computer system includes acomputer-readable medium and a processor in operable communication withthe computer-readable medium. In one embodiment, by way of example only,the processor, configures multiple resolution levels for a similaritysearch. Input similarity elements are calculated in one resolution levelfor a chunk of input data. The input similarity elements of the oneresolution level are used to find similar data in a repository of data,where similarity elements of the stored similar repository data are ofthe multiple resolution levels.

In a further embodiment, a computer program product is provided foradaptive similarity search using compatibility and inclusion ofsimilarity element resolutions in a data deduplication system using aprocessor device, in a computing environment. The computer-readablestorage medium has computer-readable program code portions storedthereon. The computer-readable program code portions include a firstexecutable portion that, configures multiple resolution levels for asimilarity search. Input similarity elements are calculated in oneresolution level for a chunk of input data. The input similarityelements of the one resolution level are used to find similar data in arepository of data, where similarity elements of the stored similarrepository data are of the multiple resolution levels.

In addition to the foregoing exemplary method embodiment, otherexemplary system and computer product embodiments are provided andsupply related advantages. The foregoing summary has been provided tointroduce a selection of concepts in a simplified form that are furtherdescribed below in the Detailed Description. This Summary is notintended to identify key features or essential features of the claimedsubject matter, nor is it intended to be used as an aid in determiningthe scope of the claimed subject matter. The claimed subject matter isnot limited to implementations that solve any or all disadvantages notedin the background.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict embodiments of the invention and are not therefore to beconsidered to be limiting of its scope, the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 is a block diagram illustrating a computing system environmenthaving an example storage device in which aspects of the presentinvention may be realized;

FIG. 2 is a block diagram illustrating a hardware structure of a datastorage system in a computer system in which aspects of the presentinvention may be realized;

FIG. 3 is a flowchart illustrating an exemplary method for adaptivesimilarity search resolution in a data deduplication system in whichaspects of the present invention may be realized;

FIG. 4 is a flowchart illustrating an exemplary method for processing aset of input data chunks with adaptive similarity search resolution inwhich aspects of the present invention may be realized;

FIG. 5 is a flowchart illustrating an exemplary method for calculatingsets of similarity element matches in which aspects of the presentinvention may be realized;

FIG. 6 is a block diagram illustrating exemplary sets of similarityelement matches in which aspects of the present invention may berealized; and

FIG. 7 is a flowchart illustrating an exemplary method for calculating aresolution for storing similarity elements in which aspects of thepresent invention may be realized.

DETAILED DESCRIPTION OF THE DRAWINGS

Data deduplication is a highly important and vibrant field in computingstorage systems. Data deduplication refers to the reduction and/orelimination of redundant data. In data deduplication, a data object,which may be a file, a data stream, or some other form of data, ispartitioned into one or more parts called chunks or blocks. In a datadeduplication process, duplicate copies of data are reduced oreliminated, leaving a minimal amount of redundant copies, or a singlecopy of the data, respectively. The goal of a data deduplication systemis to store a single copy of duplicated data, and the challenges inachieving this goal are efficiently finding the duplicate data patternsin a typically large repository, and storing the data patterns in astorage efficient deduplicated form. A significant challenge indeduplication storage systems is scaling to support very largerepositories of data. Such large repositories can reach sizes ofPetabytes (1 Petabyte=2⁵⁰ bytes) or more. Deduplication storage systemssupporting such repository sizes, must provide efficient processing forfinding duplicate data patterns within the repositories, whereefficiency is measured in resource consumption for achievingdeduplication (resources may be CPU cycles, RAM storage, persistentstorage, networking, etc.). In one embodiment, a deduplication storagesystem may be based on maintaining a search optimized index of valuesknown as fingerprints or digests, where a small fingerprint represents alarger block of data in the repository. The fingerprint values may becryptographic hash values calculated based on the blocks' data. In oneembodiment, secure hash algorithm (SHA), e.g. SHA-1 or SHA-256, whichare a family of cryptographic hash functions, may be used. Identifyingfingerprint matches, using index lookup, enables to store references todata that already exists in a repository. In one embodiment, blockboundaries may be determined based on the data itself.

To provide reasonable deduplication in this approach, the mean size ofthe data blocks based on which fingerprints are generated must belimited to smaller sizes and may not be too large. The reason being thata change of a bit within a data block will probabilistically change thedata block's corresponding fingerprint, and thus having large datablocks makes the scheme more sensitive to updates in the data ascompared to having small blocks. A typical data block size may rangefrom 4 KB to 64 KB, depending on the type of application and workload.Thus, by way of example only, small data blocks may range in sizes of upto 64 KB, and large data blocks are those data blocks having a sizelarger than 64 KB.

To support very large repositories scaling to Petabytes (e.g.,repositories scaling to at least one Petabyte=2⁵⁰ bytes), the number offingerprints to store coupled with the size of a fingerprint (rangingbetween 16 bytes and 64 bytes), becomes prohibitive. For example, for 1Petabyte of deduplicated data, 4 Kilobytes mean data block size, and 32bytes fingerprint size (e.g. of SHA-256), the storage required to storethe fingerprints is 8 Terabytes. Maintaining a search optimized datastructure for such volumes of fingerprints is difficult, and existingoptimization techniques do not scale to these sizes while maintainingperformance. For this reason, to provide reasonable performance, thesupported repositories have to be relatively small (on the order of tensof Terabytes). Even for such smaller sizes, considerable challenges andrun-time costs arise due to the large scale of the fingerprint indexesthat create a bottle-neck in deduplication processing.

In one embodiment, by way of example only, the term “similar data” maybe referred to as: for any given input data, data which is similar tothe input data is defined as data which is mostly the same (i.e. notentirely but at least 50% similar) as the input data. From looking atthe data in a binary perspective, this means that similar data is datawhere most (i.e. not entirely but at least 50% similar) of the bytes arethe same as the input data.

In one embodiment, by way of example only, the term “similarity search”may be referred to as the process of searching for data which is similarto input data in a repository of data. In one embodiment, this processmay be performed using a search structure of similarity elements, whichis maintained and searched within.

In one embodiment, by way of example only, the term “similarityelements” represents elements which may be calculated based on the dataand facilitate a global search for data which is similar to input datain a repository of data. In general, one or more similarity elements arecalculated, and represent a large (e.g. at least 4 MB) chunk of data.

Thus, as described above, the deduplication approach of the presentinvention uses a two-step process for finding data matches duringdeduplication. In the first step, a large chunk of input data (e.g. 4MB) is searched in the repository for similar (rather than identical)chunks of existing data, and the input chunk is partitioned accordinglyinto intervals, and paired with corresponding similar repositoryintervals. The similarity search structure (or “index”) used in thefirst step is compact and simple to maintain and search within, becausethe elements used for a similarity search are very compact relative tothe data they represent (e.g. 16 bytes representing 4 megabytes). In thesecond step, matches of identical data are found in the similar inputand repository intervals. This can be done by comparing the data ofthese intervals or by comparing digests of the data. This approach istermed as the similarity based deduplication approach.

At the basis of the similarity based deduplication approach there is afundamental trade-off involving the size of the data chunks representedby each of the similarity elements used in similarity search. If thechunks are large, then the similarity index is smaller, thus requiringless computational resources (i.e. less IO, storage, RAM, networking,and mutual exclusion) for working with the index (i.e. search and updateoperations). However, with large chunks, the reliability of the resultsof similarity search becomes more sensitive to the internal reorderingrate and the change rate of a workload. Namely, similarity searchbecomes less effective as the reordering rate or change rate increase.

In one embodiment, in deduplication systems that maintain a large chunksize for similarity search (e.g. 8 MB or more), to keep resourceconsumption sustainable, this can cause low effectiveness or eveninability to effectively deduplicate workloads with inherent reorderingor high change rates. There are many examples of such workloads, e.g.multiplexed backups (multiplexing is used to increase backup performanceand comply with backup window times), virtual machine workloads, andworkloads of databases or other applications using compaction ordefragmentation. In deduplication systems as those mentioned above, suchworkloads can reduce or eliminate the ability of the similarity searchprocess to find similar data, and therefore can considerably reduce theoverall deduplication effectiveness of the deduplication system. Thesetypes of workloads are very typical in user environments, and thereforethis limitation can reduce the competitiveness of a deduplicationsystem.

In a multiplexed workload a data set is read by concurrent backupprocesses and streamed into a storage system. This concurrency enhancesbackup performance, and is very common in backup environments. However,this concurrency causes internal reordering of sections of the data,between generations of data that should deduplicate with each other.Such reordering is very probable to cause the similarity elementscalculated for a given generation of a data set to be different thanthose calculated for previous generations of the data set, thus reducingor eliminating the ability to find similar data.

For multiplexed workloads, fitting the chunk size used for similaritysearch to accord with the granularity of reordering will improvededuplication. For example, if sections of 1 MB are reordered acrossgenerations of a data set, then to maximize the deduplication potential,for each 1 MB of input data a similar repository interval should befound. If, for instance, the chunk size used is 8 MB (namely, onesimilarity element for each chunk of 8 MB), then a similar 8 MB intervalis found for each 8 MB of input data, therefore likely losing 7 MB ofpotential deduplication for each input chunk. If the chunk size used issmaller, e.g. 1 MB, hence finding a similar 1 MB interval for each 1 MBof input data, then the deduplication potential is maximized in thisexample. However, reducing the chunk size entails additional resourceconsumption costs.

Workloads with high change rates and/or internal reordering are typicalfor data sets that undergo frequent updates, or undergo compaction ordefragmentation. In such workloads a given generation of a data set isconsiderably different than a previous generation of the same data set.This is very likely to cause the similarity elements calculated for agiven generation to be different than those calculated for the previousgenerations, thus reducing or eliminating the ability to find similardata.

In one embodiment, when there is higher probability of change in thesimilarity elements, due to high change rates and/or internalreordering, having a smaller chunk size enables having more similarityelements for a given size of data, which increases the probability thatsome of the similarity elements will remain the same betweengenerations. For example, by having one similarity element for an 8 MBchunk, the fate of the entire 8 MB of data depends on the stability ofthat single similarity element between generations. By lowering thechunk size, the fate of less data depends on the stability of each ofthe similarity elements, and therefore the similarity search processbecomes less sensitive to the effects of the change rate and theinternal reordering rate. However, as said, reducing the chunk sizeentails additional resource consumption costs.

Thus, a smaller chunk size for similarity search has a considerableimproving effect on deduplication, but also considerably increases theresource consumption of the operations on the similarity index. This isdenoted as the similarity resolution trade-off.

Therefore a need exists for a solution that will enable using smallerchunk sizes in similarity search, to improve deduplication effectivenessand enable deduplication of the above workload types, while keepingresource consumption low. Thus, in one embodiment, the present inventionprovides a solution of adaptive similarity search resolution in asimilarity based data deduplication system using a processor device in acomputing environment. The adaptive similarity search resolution enablesautomatic selection of an optimal similarity search chunk size for eachworkload, thus enabling usage of smaller chunk sizes where required, toimprove deduplication and enable effective deduplication of theworkloads specified above, while keeping resource consumption low. Inone embodiment, the present invention partitions the input data intochunks, and high resolution input similarity elements are calculated foreach input chunk. The high resolution input similarity elements are usedto find similar data in a repository of data using a similarity searchstructure. A resolution level is calculated for storing the inputsimilarity elements. The input similarity elements are stored in thecalculated resolution level in the similarity search structure.

In one embodiment, the present invention provides a similarity baseddeduplication process where a stream of input data is partitioned intochunks (e.g. 8 MB), and each chunk is processed in two main steps. Inthe first step a similarity search process is applied, and positions ofthe most similar reference data in the repository are found. Then, inthe second step, the input chunk is matched with the similar repositorydata, to form data matches, by either comparing digests of the data, orcomparing the data itself.

A main idea underlying the present invention, as described herein, isthat different types of workloads may be processed with differentresolutions of similarity search to produce optimal deduplication.Namely, workloads with low reordering and change rates can be processedwith low resolution similarity and produce optimal deduplication, whileworkloads with high reordering or change rates should be processed withhigh resolution similarity to produce optimal deduplication. Further,several levels of resolutions can be defined to accommodate differenttypes of workloads. For example, a low resolution can be defined to be asingle similarity element for a chunk size of 8 MB, and a highresolution can be defined to be a single similarity element for a chunksize of 1 MB. In between, the following resolutions can be defined: asingle similarity element for a chunk size of 2 MB, and for a chunk sizeof 4 MB. Thus forming 4 levels of resolution of similarity search,accommodating workloads with varying reordering and change rates. As aworkload is characterized by higher reordering or change rates, a highersimilarity resolution should be used to process it.

An additional main idea underlying the present invention is that withthe calculation method of similarity elements used in this invention,the different resolutions are compatible with each other. Namely,similarity elements of a specific resolution may be used to find similardata for which a different resolution of similarity elements wasproduced and stored. The reason for this compatibility is that for eachspecific resolution, its similarity elements are a subset of thesimilarity elements of resolutions, which are higher than that specificresolution. Consider the following example: Four resolutions aredefined—a single similarity element for chunk sizes of 1 MB, 2 MB, 4 MBand 8 MB. The similarity elements of the 2 MB resolution are a subset ofthe similarity elements of the 1 MB resolution. The similarity elementsof the 4 MB resolution are a subset of the similarity elements of the 2MB resolution, and so forth. This property is denoted herewith as thesimilarity elements inclusion property. The reason for this propertyderives from the method of calculation of similarity elements. Accordingto the calculation method of similarity elements, the similarityelements are selected by calculating either the maximal value, or theminimal value, or any repeatable selection criterion, from the rollinghash values calculated for a chunk of data. Consider for exampleselection of the maximal value. The maximal value of the rolling hashvalues of a 2 MB chunk, is one of the two maximal values of the rollinghash values calculated for the two 1 MB chunks constituting the 2 MBchunk. Therefore, the similarity element calculated for the 2 MB chunkis one of the two similarity elements calculated for the two 1 MB chunksconstituting the 2 MB chunk. Hence the inclusion property of thesimilarity elements.

Consider an input chunk of data, and a similar chunk of data alreadystored in the repository within a previous generation of the data set.Consider a first case where the similarity elements calculated for theinput chunk are of higher resolution than those of the repository chunk.Since, according to the inclusion property, the similarity elements ofthe repository chunk are a subset of the similarity elements of theinput chunk then some of the similarity elements of the input chunk willbe matched with the similarity elements of the repository chunk. Thiswill enable to consider the repository chunk in the similarity searchprocessing, and output the repository chunk as a similar repositoryinterval. Consider a second case where the similarity elementscalculated for the input chunk are of lower resolution than those of therepository chunk. Since, according to the inclusion property, thesimilarity elements of the input chunk are a subset of the similarityelements of the repository chunk then the similarity elements of theinput chunk will be matched with the similarity elements of therepository chunk. This will enable to consider the repository chunk inthe similarity search processing, and output the repository chunk as asimilar repository interval. Hence the compatibility property of thevarious resolutions of similarity search.

In one embodiment, the present invention provides an algorithm fordetermining the appropriate similarity resolution for a given workload.The algorithm of the present invention enables to use a highersimilarity resolution for workloads with high reordering and/or highchange rates, and a lower similarity resolution for workloads with lowreordering and change rates. This is achieved by the algorithm learningthe properties of the workloads using calculated statistics,distinguishing between workloads that require higher similarityresolution and workloads, which require lower similarity resolution, andapplying appropriate similarity resolution for each workload.

In one embodiment, the present invention solves a considerablelimitation of similarity based deduplication systems, which commonly cannot produce good deduplication for workloads which are difficult fordeduplication, e.g. workloads with high reordering rate or high changerate. In one embodiment, the present invention enables similarity baseddeduplication systems to effectively deduplicate difficult workloads,and improve the overall deduplication results, while keeping resourceconsumption low. Examples of such workloads are multiplexed backups,virtual machine data sets, and databases backups. Since such workloadsare typical in customer environments, this solution resolves aconsiderable limitation of similarity based deduplication systems andsignificantly improves their competitiveness.

Turning now to FIG. 1, exemplary architecture 10 of a computing systemenvironment is depicted. The computer system 10 includes centralprocessing unit (CPU) 12, which is connected to communication port 18and memory device 16. The communication port 18 is in communication witha communication network 20. The communication network 20 and storagenetwork may be configured to be in communication with server (hosts) 22,24 and storage systems, which may include storage devices 14. Thestorage systems may include hard disk drive (HDD) devices, solid-statedevices (SSD) etc., which may be configured in a redundant array ofindependent disks (RAID). The operations as described below may beexecuted on storage device(s) 14, located in system 10 or elsewhere andmay have multiple memory devices 16 working independently and/or inconjunction with other CPU devices 12. Memory device 16 may include suchmemory as electrically erasable programmable read only memory (EEPROM)or a host of related devices. Memory device 16 and storage devices 14are connected to CPU 12 via a signal-bearing medium. In addition, CPU 12is connected through communication port 18 to a communication network20, having an attached plurality of additional computer host systems 24.In addition, memory device 16 and the CPU 12 may be embedded andincluded in each component of the computing system 10. Each storagesystem may also include separate and/or distinct memory devices 16 andCPU 12 that work in conjunction or as a separate memory device 16 and/orCPU 12.

FIG. 2 is an exemplary block diagram 200 showing a hardware structure ofa data storage system in a computer system according to the presentinvention. Host computers 210, 220, 225, are shown, each acting as acentral processing unit for performing data processing as part of a datastorage system 200. The cluster hosts/nodes (physical or virtualdevices), 210, 220, and 225 may be one or more new physical devices orlogical devices to accomplish the purposes of the present invention inthe data storage system 200. In one embodiment, by way of example only,a data storage system 200 may be implemented as IBM® ProtecTIER®deduplication system TS7650G™. A Network connection 261 may be a fibrechannel fabric, a fibre channel point to point link, a fibre channelover ethernet fabric or point to point link, a FICON or ESCON I/Ointerface, any other I/O interface type, a wireless network, a wirednetwork, a LAN, a WAN, heterogeneous, homogeneous, public (i.e. theInternet), private, or any combination thereof. The hosts, 210, 220, and225 may be local or distributed among one or more locations and may beequipped with any type of fabric (or fabric channel) (not shown in FIG.2) or network adapter 261 to the storage controller 240, such as Fibrechannel, FICON, ESCON, Ethernet, fiber optic, wireless, or coaxialadapters. Data storage system 200 is accordingly equipped with asuitable fabric (not shown in FIG. 2) or network adaptor 261 tocommunicate. Data storage system 200 is depicted in FIG. 2 comprisingstorage controllers 240 and cluster hosts 210, 220, and 225. The clusterhosts 210, 220, and 225 may include cluster nodes.

To facilitate a clearer understanding of the methods described herein,storage controller 240 is shown in FIG. 2 as a single processing unit,including a microprocessor 242, system memory 243 and nonvolatilestorage (“NVS”) 216. It is noted that in some embodiments, storagecontroller 240 is comprised of multiple processing units, each withtheir own processor complex and system memory, and interconnected by adedicated network within data storage system 200. Storage 230 (labeledas 230 a, 230 b, and 230 n in FIG.2) may be comprised of one or morestorage devices, such as storage arrays, which are connected to storagecontroller 240 (by a storage network) with one or more cluster hosts210, 220, and 225 connected to each storage controller 240.

In some embodiments, the devices included in storage 230 may beconnected in a loop architecture. Storage controller 240 manages storage230 and facilitates the processing of write and read requests intendedfor storage 230. The system memory 243 of storage controller 240 storesprogram instructions and data, which the processor 242 may access forexecuting functions and method steps of the present invention forexecuting and managing storage 230 as described herein. In oneembodiment, system memory 243 includes, is in association with, or is incommunication with the operation software 250 for performing methods andoperations described herein. As shown in FIG. 2, system memory 243 mayalso include or be in communication with a cache 245 for storage 230,also referred to herein as a “cache memory”, for buffering “write data”and “read data”, which respectively refer to write/read requests andtheir associated data. In one embodiment, cache 245 is allocated in adevice external to system memory 243, yet remains accessible bymicroprocessor 242 and may serve to provide additional security againstdata loss, in addition to carrying out the operations as described inherein.

In some embodiments, cache 245 is implemented with a volatile memory andnonvolatile memory and coupled to microprocessor 242 via a local bus(not shown in FIG. 2) for enhanced performance of data storage system200. The NVS 216 included in data storage controller is accessible bymicroprocessor 242 and serves to provide additional support foroperations and execution of the present invention as described in otherfigures. The NVS 216, may also referred to as a “persistent” cache, or“cache memory” and is implemented with nonvolatile memory that may ormay not utilize external power to retain data stored therein. The NVSmay be stored in and with the cache 245 for any purposes suited toaccomplish the objectives of the present invention. In some embodiments,a backup power source (not shown in FIG. 2), such as a battery, suppliesNVS 216 with sufficient power to retain the data stored therein in caseof power loss to data storage system 200. In certain embodiments, thecapacity of NVS 216 is less than or equal to the total capacity of cache245.

Storage 230 may be physically comprised of one or more storage devices,such as storage arrays. A storage array is a logical grouping ofindividual storage devices, such as a hard disk. In certain embodiments,storage 230 is comprised of a JBOD (Just a Bunch of Disks) array or aRAID (Redundant Array of Independent Disks) array. A collection ofphysical storage arrays may be further combined to form a rank, whichdissociates the physical storage from the logical configuration. Thestorage space in a rank may be allocated into logical volumes, whichdefine the storage location specified in a write/read request.

In one embodiment, by way of example only, the storage system as shownin FIG. 2 may include a logical volume, or simply “volume,” may havedifferent kinds of allocations. Storage 230 a, 230 b and 230 n are shownas ranks in data storage system 200, and are referred to herein as rank230 a, 230 b and 230 n. Ranks may be local to data storage system 200,or may be located at a physically remote location. In other words, alocal storage controller may connect with a remote storage controllerand manage storage at the remote location. Rank 230 a is shownconfigured with two entire volumes, 234 and 236, as well as one partialvolume 232 a. Rank 230 b is shown with another partial volume 232 b.Thus volume 232 is allocated across ranks 230 a and 230 b. Rank 230 n isshown as being fully allocated to volume 238—that is, rank 230 n refersto the entire physical storage for volume 238. From the above examples,it will be appreciated that a rank may be configured to include one ormore partial and/or entire volumes. Volumes and ranks may further bedivided into so-called “tracks,” which represent a fixed block ofstorage. A track is therefore associated with a given volume and may begiven a given rank.

The storage controller 240 may include a data duplication module 255, asimilarity index module 257, a similarity elements match setscalculation module 259, and an adaptive similarity search module 260.The data duplication module 255, the similarity index module 257, thesimilarity elements match sets calculation module 259, and the adaptivesimilarity search module 260 may work in conjunction with each and everycomponent of the storage controller 240, the hosts 210, 220, 225, andstorage devices 230. The data duplication module 255, the similarityindex module 257, the similarity elements match sets calculation module259, and the adaptive similarity search module 260 may be structurallyone complete module or may be associated and/or included with otherindividual modules. The data duplication module 255, the similarityindex module 257, the similarity elements match sets calculation module259, and the adaptive similarity search module 260 may also be locatedin the cache 245 or other components.

The storage controller 240 includes a control switch 241 for controllingthe fiber channel protocol to the host computers 210, 220, 225, amicroprocessor 242 for controlling all the storage controller 240, anonvolatile control memory 243 for storing a microprogram (operationsoftware) 250 for controlling the operation of storage controller 240,data for control, cache 245 for temporarily storing (buffering) data,and buffers 244 for assisting the cache 245 to read and write data, acontrol switch 241 for controlling a protocol to control data transferto or from the storage devices 230, the data duplication module 255, thesimilarity index module 257, and the similarity search module 259, inwhich information may be set. Multiple buffers 244 may be implementedwith the present invention to assist with the operations as describedherein. In one embodiment, the cluster hosts/nodes, 210, 220, 225 andthe storage controller 240 are connected through a network adaptor (thiscould be a fibre channel) 260 as an interface i.e., via at least oneswitch called “fabric.”

In one embodiment, the host computers or one or more physical or virtualdevices, 210, 220, 225 and the storage controller 240 are connectedthrough a network (this could be a fibre channel) 260 as an interfacei.e., via at least one switch called “fabric.” In one embodiment, theoperation of the system shown in FIG. 2 will be described. Themicroprocessor 242 may control the memory 243 to store commandinformation from the host device (physical or virtual) 210 andinformation for identifying the host device (physical or virtual) 210.The control switch 241, the buffers 244, the cache 245, the operatingsoftware 250, the microprocessor 242, memory 243, NVS 216, dataduplication module 255, the similarity index module 257, the similarityelements match sets calculation module 259, and the adaptive similaritysearch module 260 are in communication with each other and may beseparate or one individual component(s). Also, several, if not all ofthe components, such as the operation software 250 may be included withthe memory 243. Each of the components within the devices shown may belinked together and may be in communication with each other for purposessuited to the present invention. As mentioned above, the dataduplication module 255, the similarity index module 257, the similarityelements match sets calculation module 259, and the adaptive similaritysearch module 260 may also be located in the cache 245 or othercomponents. As such, the data duplication module 255, the similarityindex module 257, the similarity elements match sets calculation module259, and the adaptive similarity search module 260 maybe used as needed,based upon the storage architecture and users preferences.

FIG. 3 is a flowchart illustrating an exemplary method 300 for adaptivesimilarity search resolution in a data deduplication system in whichaspects of the present invention may be realized. The method 300 begins(step 302). The method 300 partitions input data into data chunks (step304). The input data may be partitioned into fixed sized data chunks.The method 300 calculates, for each of the data chunks, similarityelements (e.g., high resolution similarity elements) (step 306). Themethod 300 uses the input similarity elements (e.g., high resolutionsimilarity elements) to find similar data in a repository of data usinga similarity search structure (e.g., a similarity index module and asimilarity elements match sets calculation module, see FIG. 2 259 and/or260) (step 308). A resolution level is calculated for storing the inputsimilarity elements (step 310). The method 300 stores the inputsimilarity elements in the calculated resolution level in the similaritysearch structure (e.g., in the similarity index module and/or thesimilarity elements match sets calculation module (step 312). The method300 ends (step 314).

In one embodiment, the present invention provides an algorithm fordeduplicating input data by dynamically determining the appropriatesimilarity resolution. In one embodiment, several levels of similarityresolution are defined. In one embodiment, four levels of resolution aredefined, as follows: a single similarity element for chunk sizes of 1MB, 2 MB, 4 MB and 8 MB.

In one embodiment, the algorithm of the present invention associateswith each input stream of data the following variable properties. 1) Thecurrent level of resolution for storing similarity elements of the inputstream. This property is denoted as the current resolution levelproperty, and is initialized with the value of the highest level ofresolution. 2) The average size of the sets of similarity elementmatches of the input stream. The calculation of this property isspecified in the following. 3) The aggregated deduplication ratio of theinput stream. The calculation of this property is specified in thefollowing.

In one embodiment, an input stream of data is partitioned into chunks(e.g. of size 8 MB), and the chunks are grouped into chunk sets of apredefined size. Each one of the chunks is processed for deduplication.Determination of a resolution level for storage of the similarityelements calculated for the input chunks, and storage of thesesimilarity elements in accordance with the determined resolution, isdone for each chunk set.

FIG. 4 is a flowchart illustrating an exemplary method 400 forprocessing a set of input chunks with adaptive similarity searchresolution in which aspects of the present invention may be realized.The method 400 begins (step 402). The method 400 receives as input a setof input chunks within an input stream of data (step 406). The inputchunks are produced by partitioning the input stream of data into chunks(e.g., large fixed size chunks). The method 400 determines if there areadditional input chunks (step 408). If no, the method 400 calculates aresolution of similarity elements to be stored in a repository for thecurrent set of input chunks using an aggregated deduplication ratio ofan input stream and the average size of sets of similarity elementsmatches of an input stream (step 430). The method then stores thesimilarity elements of the input chunks of the current input chunk setin accordance with the calculated resolution (step 432). The method 400ends (step 434). However, if there are additional input chunks at step408, the method 400 calculates similarity elements of the highestresolution for the input chunk (step 410). The method 400 searches thehighest resolution similarity elements of the input chunk in asimilarity search structure (using the similarity search structure ofstep 414) and obtains matching repository similarity elements (step412). The method 400 constructs sets of similarity element matches fromthe obtained matches (step 416). The method 400 adds the sizes of theconstructed sets of similarity element matches to an aggregated averagessize of the sets of similarity element matches of the input stream(steps 418 and 420). The method 400 calculates the repository dataintervals that are most similar to input data chunk, based on the setsof similarity element matches (step 422). The method 400 matches theinput chunk with the similar repository intervals, either by comparingdigests of the data and/or comparing the data itself, to form datamatches (step 424). The method 400 adds total sizes of the matched andunmatched portions of the input chunk to an aggregated deduplicationratio of the input stream (steps 426 and 428). The method 400 thenreturns to step 408.

In one embodiment, similarity elements of the highest resolution arecalculated for an input chunk. It should be noted that there is noadditional computational cost in calculating the highest resolutionsimilarity elements, relative to any other resolution. This is becausethe calculation of the rolling hash values (i.e. a hash value for eachsmall block, e.g. of size 64 bytes, at each byte offset) is required asbasis for calculating the similarity elements for any output resolution,and this calculation is typically the major resource consumer. The highresolution similarity elements of the input chunk are searched in asimilarity search structure, and matching repository similarity elementsare obtained. The similarity element matches that will be formed areupper bound by the resolution of the repository similarity elements thatare found as matching.

In one embodiment, sets of similarity element matches are constructedfrom the obtained matches. A similarity element match is a pair ofmatching input and repository similarity elements, where each one of thesimilarity elements has a value and a position in the input and in therepository data respectively. A set of similarity element matchescontains similarity element matches of a similar angle. An angle of asimilarity element match is defined as the difference between theposition in the repository data of the repository similarity element ofthe pair, and the position in the input data of the input similarityelement of the pair. Two angles are defined as similar if theirdifference does not exceed a predefined threshold. The algorithm forcalculating sets of similarity element matches (see FIG. 5 below) inputsa sequence of similarity element matches, and sorts the sequence by theposition of the matches in the input data. The algorithm scans theordered sequence of similarity element matches from first to last, andadds similarity element matches to a current set as long as the angle ofthe next match is similar to an average angle of the matches alreadyincluded in the set. If the angle of the next match is not similar tothe average angle of the matches in the current set, then the currentset is closed, and a new current set is created holding the next match.This process continues until there are no additional similarity elementmatches to process.

The sizes of the constructed sets of similarity element matches areadded to an aggregated average size of the sets of similarity elementmatches of the input stream (see FIG. 6 below). The repository dataintervals, which are most similar to the input data chunk, arecalculated based on the sets of similarity element matches.Specifically, each set of similarity element matches serves as basis fordetermining the boundaries in the input data chunk and in the repositorydata, of the similar input and repository sub-intervals. The input chunkis matched with the similar repository intervals, either by comparingdigests of the data or comparing the data itself, to form data matches.The total sizes of the matched and unmatched portions of the input chunkare added to an aggregated deduplication ratio of the input stream. Thededuplication ratio is defined as the total size of the portions of theinput data covered by matches with repository data out of the total sizeof the input data.

When deduplication processing of the input chunks in an input chunk setis complete, the aggregated deduplication ratio of the input stream andthe average size of the sets of similarity element matches of the inputstream are used to calculate the resolution of similarity elements to bestored in the repository for the chunks in the current chunk set. Thehigh resolution similarity elements of the input chunks in a currentchunk set are stored in memory, until a determination of the appropriateresolution for storage of similarity elements is made, and then theappropriate resolution of similarity elements is extracted from thesimilarity elements in memory and stored in the repository. Lastly, thesimilarity elements of the chunks in the current chunk set are removedfrom memory.

In one embodiment, the present invention provides a method forcalculating a resolution for storing similarity elements (see FIG. 7below). In one embodiment, if the aggregated deduplication ratio issufficiently good, e.g., not lower than a predefined threshold, thenthere are two cases. If the average size of the sets of similarityelement matches is not lower than 2, then the storage resolution of theinput stream is decreased by one level. The reason is that a value of 2or larger implies that there is redundancy in the current resolutionlevel of the similarity elements. The resolution level can be reducedonly if it is higher than the lowest resolution level, otherwise itremains as is. Alternatively, if the average size of the sets ofsimilarity element matches is lower than 2, then no change is made inthe storage resolution of the input stream. The reason is that a valuelower than 2 implies that there is minimal or no redundancy in thecurrent resolution level of the similarity elements, and therefore thecurrent resolution level fits the current input stream. If theaggregated deduplication ratio is not sufficiently good, e.g., lowerthan a predefined threshold, then the storage resolution of the inputstream is increased by one level. The resolution level can be increasedonly if it is lower than the highest resolution level, otherwise itremains as is.

In one embodiment, the present invention enables similarity baseddeduplication systems to effectively deduplicate workloads that aredifficult for deduplication, e.g., workloads with high reordering rateor high change rate, while keeping resources consumption low. Examplesof such workloads are multiplexed backups, virtual machine data sets,and databases backups. Since such workloads are typical in customerenvironments, this solution resolves a considerable limitation ofsimilarity based deduplication systems and significantly improves theircompetitiveness.

FIG. 5 is a flowchart illustrating an exemplary method 500 forcalculating sets of similarity element matches in which aspects of thepresent invention may be realized. The method 500 begins (step 502). Themethod 500 inputs a sequence of similarity element matches (step 504),and sorts the sequence by the position of the matches in the input data(step 506). The method 500 determines if there is an additionalsimilarity element match (step 508). If no, the method 500 closes thecurrent set of similarity element matches and adds this set to theoutput sets (step 520), and outputs the sets of similarity matches (step522). The method 500 ends (step 524). If there is an additionalsimilarity element match at step 508, the method 500 determines if thereis an open set of similarity element matches (step 510). If no, themethod 500 creates a new set of similarity element matches (step 512)and then adds the current similarity element match to the current set ofsimilarity matches (step 518). If yes from step 510, the method 500determines if the angle of the current match is similar to the averageangle of the matches already included the set of similarity elementmatches (step 516). If no, the method 500 closes the current set ofsimilarity element matches and adds this set to the output sets (step514), and continues to step 512. If yes from step 516, the method 500moves to step 518 and adds the current similarity element match to thecurrent set of similarity matches (step 518).

FIG. 6 is a block diagram illustrating exemplary sets of similarityelement matches in which aspects of the present invention may berealized. In one embodiment, the sizes of the constructed sets ofsimilarity element matches are added to an aggregated average size ofthe sets of similarity element matches of the input stream. FIG. 6illustrates repository data 602 (shown in FIG. 6 as 602A-D) and inputdata (shown in FIG. 6 as 604A-D). FIG. 6 exemplifies 4 cases: In case Athere is a single set of similarity element matches 606; in case B thereare 2 sets 606 and 608; in case C there are 4 sets 606, 608, 610, and612; and in case D there are 8 sets 606, 608, 610, 612, 614, 616, 618,and 620. The average sizes of the sets 606, 608, 610, 612, 614, 616,618, and 620 for each case are as follows: in case A—8; in case B—4; incase C—2; and in case D—1.

FIG. 7 is a flowchart illustrating an exemplary method 700 forcalculating a resolution for storing similarity elements in whichaspects of the present invention may be realized. The method 700 begins(step 702). The method 700 determines if the aggregated deduplicationratio is not lower than a predefined threshold (step 704, using step 712which is an aggregated deduplication ratio of the input stream). If no,the method 700 increases the storage resolution of the input stream byone level, if the current resolution level is lower than the highestresolution level (step 706, using step 708 which is the currentresolution for storing the similarity elements of the input stream). Ifyes from step 704, the method 700 determines if the average size of aset of similarity element matches is not lower than 2 (step 714 usingstep 716 which is an average size of the sets of similarity elementmatches of the input stream). If yes, the method 700 decreases thestorage resolution of the input stream by one level, if the currentresolution level is higher than the lowest resolution level (step 710using step 708). If no, the method ends (step 718).

In one embodiment, the present invention provides a method, system,and/or computer program product for calculating an appropriatesimilarity resolution level for an input workload, based on the averagesize of the sets of similarity element matches and the aggregateddeduplication ratio for adaptive similarity search resolution in a datadeduplication system using a processor device in a computingenvironment. Input data is partitioned into data chunks. High resolutioninput similarity elements are calculated for each input chunk. The highresolution input similarity elements are used to find similar data in arepository of data using a similarity search structure. A resolutionlevel is calculated for storing the input similarity elements. The inputsimilarity elements are stored in the calculated resolution level in thesimilarity search structure. In one embodiment, a resolution level iscalculated for storing the input similarity elements based on calculatedsets of similarity element matches and on a calculated deduplicationratio. The calculated resolution level for storing the input similarityelements is defined to be between a highest resolution level and alowest resolution level, where each one of the resolution levels iscompatible for finding matches with each one of the other resolutionlevels. An average size of the sets of similarity element matches iscalculated, and used to determine the resolution level for storing theinput similarity elements. A set of similarity element matches isdefined to include similarity element matches with a similar angle,where an angle of a similarity element match is the difference betweenits position in the repository data and its position in the input data,and where two angles are considered as similar if their difference doesnot exceed a predefined threshold. An aggregated deduplication ratio iscalculated as the total size of the portions of the input chunks coveredby data matches out of the total size of the chunks, and the aggregateddeduplication ratio is used to determine the resolution level forstoring the input similarity elements. The storage resolution level ofthe similarity elements is decreased if the aggregated deduplicationratio is not lower than a predefined threshold and the average size ofthe sets of similarity element matches is not lower than two and thecurrent resolution level is higher than the lowest resolution level. Thestorage resolution level of the similarity elements is increased if theaggregated deduplication ratio is lower than a predefined threshold andthe current resolution level is lower than the highest resolution level.

In one embodiment, the present invention provides similarity elementresolutions having properties of compatibility and inclusion. In oneembodiment, the present invention provides a data deduplication methodwith adaptive similarity search, including configuring resolution levelsfor similarity search, calculating input similarity elements in oneresolution level for a chunk of input data, using the input similarityelements of the one resolution level to find similar data in arepository of data, where similarity elements of the stored similarrepository data are of a plurality of resolution levels.

In one embodiment, the present invention defines the resolution levelsto be between a highest resolution level and a lowest resolution level,and configures the similarity elements of each one of the resolutionlevels to be a subset of the similarity elements of each one of theresolution levels that are higher than the one resolution level.

In one embodiment, the present invention calculates similarity elementsbased on the maximum values or minimum values of rolling hash valuescalculated for chunks of input data.

In one embodiment, the present invention calculates a resolution levelfor the similarity elements of an input chunk based on calculated setsof similarity element matches and on a calculated deduplication ratio.

In one embodiment, the present invention stores the input similarityelements in the calculated resolution level in a similarity searchstructure, and uses the similarity search structure to find similarityelements of similar repository data.

In one embodiment, the present invention calculates an average size ofthe sets of similarity element matches, and uses said average size todetermine the resolution level of the input similarity elements.

In one embodiment, the present invention calculates an aggregateddeduplication ratio as the total size of the portions of the inputchunks covered by data matches out of the total size of the chunks, anduses said aggregated deduplication ratio to determine the resolutionlevel of the input similarity elements.

In one embodiment, the present invention decreases the similarityelements resolution level if the aggregated deduplication ratio is notlower than a predefined threshold and the average size of the sets ofsimilarity element matches is not lower than two and the currentresolution level is higher than the lowest resolution level; andincreases the similarity elements resolution level if the aggregateddeduplication ratio is lower than a predefined threshold and the currentresolution level is lower than the highest resolution level.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that may contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wired, optical fiber cable, RF, etc., or any suitable combination of theforegoing. Computer program code for carrying out operations for aspectsof the present invention may be written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Java, Smalltalk, C++ or the like and conventionalprocedural programming languages, such as the “C” programming languageor similar programming languages. The program code may execute entirelyon the user's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention have been described above withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems) and computer program products according toembodiments of the invention. It will be understood that each block ofthe flowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, may beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that may direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer, other programmable data processing apparatus, orother devices to cause a series of operational steps to be performed onthe computer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the above figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, may be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

What is claimed is:
 1. A method for adaptive similarity search usingcompatibility and inclusion of similarity element resolutions in a datadeduplication system using a processor device in a computingenvironment, comprising: configuring a plurality of resolution levelsfor a similarity search; calculating input similarity elements in acertain one of the plurality of resolution levels for a chunk of inputdata, wherein each resolution level is determined by identifying anumber of the input similarity elements calculated in relation to a sizeof the chunk of input data; using the input similarity elements of thecertain one of the plurality of resolution levels to find similar datain a repository of data where similarity elements of the stored similarrepository data are of the plurality of resolution levels, for locatingand reducing duplicate data within the data deduplication system;defining the plurality of resolution levels to be between a highestresolution level and a lowest resolution level; configuring thesimilarity elements of each one of the plurality of resolution levelswhich are lower than the certain one of the plurality of resolutionlevels to be a subset of the similarity elements of each one of theplurality of resolution levels which are higher than the certain one ofthe plurality of resolution levels; calculating an aggregateddeduplication ratio as a total size of input chunks of data receivedover a certain time period covered by matches with repository data outof the total size of the input chunks relative to a total size of outputdata subsequent to deduplicating all of the input chunks of datareceived over the certain time period; and using the aggregateddeduplication ratio to determine the resolution level of the inputsimilarity elements by decreasing the similarity elements resolutionlevel if the aggregated deduplication ratio is not lower than apredefined threshold and an average size of calculated sets ofsimilarity element matches is not lower than two and a currentresolution level is higher than the lowest resolution level.
 2. Themethod of claim 1, further including calculating the similarity elementsbased on one of maximum values and minimum values of rolling hash valuescalculated for the input chunks.
 3. The method of claim 1, furtherincluding calculating a resolution level for the similarity elements ofan input chunk based on the calculated sets of similarity elementmatches and on a calculated deduplication ratio.
 4. The method of claim3, further including storing the input similarity elements in thecalculated resolution level in a similarity search structure, and usingthe similarity search structure to find similarity elements of similarrepository data.
 5. The method of claim 3, further including calculatingthe average size of the sets of similarity element matches, and usingthe average size to determine the resolution level of the inputsimilarity elements.
 6. The method of claim 5, further includingincreasing the similarity elements resolution level if the aggregateddeduplication ratio is lower than a predefined threshold and the currentresolution level is lower than the highest resolution level.
 7. A systemfor adaptive similarity search using compatibility and inclusion ofsimilarity element resolutions in a data deduplication system of acomputing environment, the system comprising: the data deduplicationsystem; a repository operating in the data deduplication system; amemory in the data deduplication system; a similarity search structurein association with the memory in the data deduplication system; and atleast one processor device operable in the computing storage environmentfor controlling the data deduplication system, wherein the at least oneprocessor device: configuring a plurality of resolution levels for asimilarity search, calculating input similarity elements in a certainone resolution of the plurality of levels for a chunk of input data,wherein each resolution level is determined by identifying a number ofthe input similarity elements calculated in relation to a size of thechunk of input data, using the input similarity elements of the certainone of the plurality of resolution levels to find similar data in therepository of data where similarity elements of the stored similarrepository data are of the plurality of resolution levels, for locatingand reducing duplicate data within the data deduplication system,defining the plurality of resolution levels to be between a highestresolution level and a lowest resolution level, configuring thesimilarity elements of each one of the plurality of resolution levelswhich are lower than the certain one of the plurality of resolutionlevels to be a subset of the similarity elements of each one of theplurality of resolution levels which are higher than the certain one ofthe plurality of resolution levels, calculating an aggregateddeduplication ratio as a total size of input chunks of data receivedover a certain time period covered by matches with repository data outof the total size of the input chunks relative to a total size of outputdata subsequent to deduplicating all of the input chunks of datareceived over the certain time period; and using the aggregateddeduplication ratio to determine the resolution level of the inputsimilarity elements by decreasing the similarity elements resolutionlevel if the aggregated deduplication ratio is not lower than apredefined threshold and an average size of calculated sets ofsimilarity element matches is not lower than two and a currentresolution level is higher than the lowest resolution level.
 8. Thesystem of claim 7, wherein the at least one processor device thatcalculates the similarity elements based on one of maximum values andminimum values of rolling hash values calculated for the input chunks.9. The system of claim 7, wherein the at least one processor devicecalculates a resolution level for the similarity elements of an inputchunk based on the calculated sets of similarity element matches and ona calculated deduplication ratio.
 10. The system of claim 9, wherein theat least one processor device stores the input similarity elements inthe calculated resolution level in the similarity search structure, andusing the similarity search structure to find similarity elements ofsimilar repository data.
 11. The system of claim 9, wherein the at leastone processor device calculates the average size of the sets ofsimilarity element matches, and uses the average size to determine theresolution level of the input similarity elements.
 12. The system ofclaim 11, wherein the at least one processor device that increases thesimilarity elements resolution level if the aggregated deduplicationratio is lower than a predefined threshold and the current resolutionlevel is lower than the highest resolution level.
 13. A computer programproduct for adaptive similarity search using compatibility and inclusionof similarity element resolutions in a data deduplication system using aprocessor device in a computing environment, the computer programproduct comprising a non-transitory computer-readable storage mediumhaving computer-readable program code portions stored therein, thecomputer-readable program code portions comprising: an executableportion that configures a plurality of resolution levels for asimilarity search; an executable portion that calculates inputsimilarity elements in a certain one of the plurality of resolutionlevels for a chunk of input data, wherein each resolution level isdetermined by identifying a number of the input similarity elementscalculated in relation to a size of the chunk of input data; anexecutable portion that uses the input similarity elements of thecertain one of the plurality of resolution levels to find similar datain a repository of data where similarity elements of the stored similarrepository data are of the plurality of resolution levels, for locatingand reducing duplicate data within the data deduplication system; anexecutable portion that defines the plurality of resolution levels to bebetween a highest resolution level and a lowest resolution level; anexecutable portion that configures the similarity elements of each oneof the plurality of resolution levels which are lower than the certainone of the plurality of resolution levels to be a subset of thesimilarity elements of each one of the plurality of resolution levelswhich are higher than the certain one of the plurality of resolutionlevels; an executable portion that calculates an aggregateddeduplication ratio as a total size of input chunks of data receivedover a certain time period covered by matches with repository data outof the total size of the input chunks relative to a total size of outputdata subsequent to deduplicating all of the input chunks of datareceived over the certain time period; and an executable portion thatuses the aggregated deduplication ratio to determine the resolutionlevel of the input similarity elements by decreasing the similarityelements resolution level if the aggregated deduplication ratio is notlower than a predefined threshold and an average size of calculated setsof similarity element matches is not lower than two and a currentresolution level is higher than the lowest resolution level.
 14. Thecomputer program product of claim 13, further including an executableportion that that calculates the similarity elements based on one ofmaximum values and minimum values of rolling hash values calculated forthe input chunks.
 15. The computer program product of claim 13, furtherincluding an executable portion that that calculates a resolution levelfor the similarity elements of an input chunk based on the calculatedsets of similarity element matches and on a calculated deduplicationratio.
 16. The computer program product of claim 15, further includingan executable portion that that stores the input similarity elements inthe calculated resolution level in a similarity search structure, andusing the similarity search structure to find similarity elements ofsimilar repository data.
 17. The computer program product of claim 15,further including an executable portion that that calculates the averagesize of the sets of similarity element matches, and uses the averagesize to determine the resolution level of the input similarity elements.18. The computer program product of claim 17, further including anexecutable portion that increases the similarity elements resolutionlevel if the aggregated deduplication ratio is lower than a predefinedthreshold and the current resolution level is lower than the highestresolution level.